A Full-Mission Evaluation of A Computational Model of Situational Awareness

Although use of the term situational awareness (SA) assists researchers in creating more fruitful environments for pilots to operate in, its true potential as a psychological construct remains untapped until a valid means of a priori predicting SA becomes available. Shively, Brickner and Silbiger (1997) proposed a computational model of SA (CSA) that seeks to do just that, and the current line of research is a series of studies aimed at validating that model. Originally developed for the Man-machine Integration Design and Analysis System (MIDAS), the CSA model is comprised of two features: situational elements and situation-sensitive higher-order nodes. Situational elements comprise what is known/perceived about the environment (e.g. tank1 or waypoint3). Each is associated with a particular higher-order node, and as a group define the situation. Higher-order nodes are semantically related groups of SE's (e.g. threats or navigation) that are weighted based on their importance in the situation. One original aspect of the CSA model is the differentiation between perceived, actual and error SA, a proportion of which produces the operator's predicted SA. Initial validation studies using low-fidelity tasks supported the predictions of the model. Preliminary data analysis of a mid-fidelity, full-mission task completed in the Rotorcraft Part-Task Laboratory (RPTL) at NASA Ames also indicates support for predictions of the model. Introduction Computational Model of Situational Awareness In an effort to accurately predict SA prior to a task and across environments, Shively, Brickner and Silbinger (1997) created a computational model of situational awareness. Initially conceived for the Manmachine Integration Design and Analysis System (MIDAS) (Smith & Tyler, 1997), the computational model of situational awareness is simply the ratio of the operator's relevant knowledge to the information needed for the mission task. Further, this model makes a distinction between actual, perceived and error SA. Goal of Research The goals of this line of research were threefold: • To develop a computational, clearly defined, predictive model of SA. • distinguish between perceived and actual SA. • validate this model in simulation. Situational Awareness Model The SA model as currently instantiated in MIDAS is comprised of two essential features, situational elements and situation-sensitive nodes. These two features will be briefly described, but for a more detailed description of the model see Shively et al., 1997. Situational Elements (SE's) are relevant information in the environment that define the situation. These include such things as other aircraft, obstacles, waypoints, ownship parameters, etc. The operator receives these elements mostly through perception; however, they are also received through paths such as experience or a pre-flight briefing. Each SE has a mathematical weight attached to it based on its importance in the situation. In addition to its weight, each SE has a mathematical value associated with it based upon one of four levels of awareness. The four levels of awareness (detection, recognition, identification, comprehension) are a quantification of the operator's perception of the SE. Situation-sensitive nodes are semantically related collections of situational elements. The nodes are defined by what is important in the situation and are weighted by the overall importance of the node in determining the level of SA. Thus, in the current study; one node is Navigation and it is comprised of the SE's flight path, all the waypoints, landmarks, and the accident site. Because visual navigation is so important to the task, the weight on the node is .5. If the situation changes then the weights on the nodes, or the nodes themselves, may change to accurately reflect the ideal SA. One original facet of the model is the differentiation of perceived, actual and error SA. Perceived SA is what operators work with...the level of SA they think they have. However, it may include errors in perception or identification. Error SA is a computation of SE's that are misperceived or misidentified. Actual SA is differentiated from perceived SA by including factors for unknown SE's in the computation and then subtracting the error component. Course of research A series of studies was conducted at NASA Ames Research Center in an attempt to validate this model and to see if the predictions of operator SA were equal to generally accepted measures of operator SA. Each of the three studies will be described. Study 1 Star Cruiser The goal of the initial study was simply to test the hypothesis that situational awareness could be broken down into actual and perceived SA. Based on the assumption that the subjective SART measurement (Taylor, 1990) is a valid measure of perceived SA and that the objective SAGAT measurement (Endlsey, 1995) is a valid measure of actual SA, this study sought to show that operators can be induced to believe their SA is higher than it really is. 10 college students performed 7 trials on the Star Cruiser software program. Each trial required the participant to fly a simulated spacecraft through a galaxy and perform various mining and exploring tasks. After each trial half of the participants were told they were doing extremely well and the other half were given neutral feedback. Both SART and SAGAT were collected on each trial in addition to various performance measures. As can be seen from some of the results in Figure 1, as time progressed the perceived SA (SART SA) of the neutral group fell over time while the perceived SA of the positive feedback group rose. While these differences did not reach significance, the trends were in the predicted direction. Figure 1. SART SA scores by function of trial order Study 2 Window/PANES This study used the Workload/PerformANcE Simulation software (Window/PANES, NASA Ames Research Center, 1989) to create a low-fidelity cockpit simulation. The study was based on a oneway, (high predicted SA vs. low predicted SA) within-subjects design. Ten GA pilots completed six pairs 60 65 70 75 80 85 90 95 100